User terminal device and controlling method thereof

ABSTRACT

A user terminal device is provided. The device includes a display configured to display a screen including an object, a detector configured to detect a user drawing input which is input on the object or in an area surrounding the object, a controller configured to, in response to the user drawing input being detected, execute a program corresponding to the object and display, on the display, a screen where the user drawing input is processed as an input of the program. Accordingly, the operations of the user terminal device may be controlled conveniently.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority from Korean Patent Application Nos.10-2013-0046991, filed in the Korean Intellectual Property Office onApr. 26, 2013, and 10-2013-0097820, filed in the Korean IntellectualProperty Office on Aug. 19, 2013, the disclosures of which areincorporated herein by reference in their entirety.

BACKGROUND

1. Field

Apparatuses and methods consistent with exemplary embodiments relate toa user terminal device and a controlling method thereof, and moreparticularly, to a user terminal device which displays a screenincluding an object and performs a control operation corresponding to auser manipulation regarding the object and a controlling method thereof.

2. Description of the Related Art

With the development of electronic technologies, various types ofelectronic apparatuses have been developed and distributed. Inparticular, a lot of portable user terminal devices such as mobilephones, tablet personal computers (PCs), laptop PCs, etc. have beenwidely used these days.

Such user terminal devices may provide services to users using variousapplications. A user may execute a desired application by selecting anicon displayed on a home screen or an application icon screen.

Accordingly, there is a need for a user interaction technology whichallows a user to use a desired service more conveniently and easily.

SUMMARY

Exemplary embodiments address at least the above problems and/ordisadvantages and other disadvantages not described above. Also,exemplary embodiments are not required to overcome the disadvantagesdescribed above, and an exemplary embodiment may not overcome any of theproblems described above.

One or more exemplary embodiments relates to a user terminal devicecapable of displaying a screen including an object, detecting a usermanipulation regarding the object in various ways, and executing atleast one control operation conveniently and quickly and a controllingmethod thereof.

According to an aspect of an exemplary embodiment, there is a provided auser terminal device including a display configured to display a screenincluding an object, a detector configured to detect a user drawinginput which is input on the object or in an area surrounding the object,and a controller configured to, in response to the user drawing inputbeing detected, execute a program corresponding to the object anddisplay, on the display, a screen where the user drawing input isprocessed as an input of the program.

Herein, the controller may be further configured to, in response to theuser drawing input including text, display, on the display, a screenwhere the text is processed as an input of the program.

The controller may be further configured to, in response to the objectbeing a search widget, perform a search regarding the recognized textand display, on the display, a result of the search.

The controller may be further configured to, in response to the userdrawing input including a symbol, display, on the display, a screenwhere the symbol is processed as an input of the program.

The controller may be further configured to, in response to the objectbeing an icon matched with a mail program, display, on the display, amail writing screen to transmit mail to a mail address corresponding tothe text.

The controller may be further configured to, in response to the objectbeing an icon matched with a messenger program, display, on the display,messenger contents which are exchanged with another user correspondingto the text.

The user drawing input may be input through a touch-and-dragmanipulation using a user's body or a pen.

According to an aspect of another exemplary embodiment, there isprovided a user terminal device including a display configured todisplay a screen including a plurality of objects, a detector configuredto detect a user manipulation on the screen, and a controller configuredto, in response to a connection line between a first object and a secondobject from among the plurality of objects being drawn according to theuser manipulation, perform a first control operation corresponding tothe first object and a second control operation corresponding to thesecond object in association with each other.

Herein, the controller may be further configured to display a firstscreen by executing a first program corresponding to the first object,and in response to a user manipulation being performed on the firstscreen, execute a second program corresponding to the second object anddisplay a second screen which reflects a result according to the usermanipulation.

Also, the controller may be further configured to determine a contentwhich is displayed on the first screen according to a type of the secondprogram.

According to an aspect of another exemplary embodiment, there isprovided a controlling method of a user terminal device includesdisplaying a first screen including an object, detecting a user drawinginput which is input on the object or in an area surrounding the object,and executing, in response to the user drawing input being detected, aprogram corresponding to the object and displaying a second screen wherethe user drawing input is processed as an input of the program.

The displaying the second screen may include, in response to the userdrawing input including text, displaying a screen where the text isprocessed as an input of the program.

The displaying the second screen may include, in response to the objectbeing a search widget, performing a search regarding the text anddisplaying a result of the search.

The displaying the screen may include, in response to the user drawinginput including a symbol, displaying a screen where the symbol isprocessed as an input of the program.

The displaying the second screen may include, in response to the objectbeing an icon matched with a mail program, displaying a mail writingscreen to transmit mail to a mail address corresponding to the text.

The displaying the second screen may include, in response to the objectbeing an icon matched with a messenger program, displaying messengercontents which are exchanged with another user corresponding to thetext.

The displaying the second screen may further include, in response to theuser drawing input being detected while a predetermined condition issatisfied, displaying the second screen according to the user drawinginput, and in response to the user drawing input being detected whilethe condition is not satisfied, changing the first screen by page unitaccording to a direction of the user drawing input.

According to an aspect of another exemplary embodiment, there isprovided a controlling method of a user terminal device includingdisplaying a screen including a plurality of objects, detecting a usermanipulation on the screen, and in response to a connection line betweena first object and a second object from among the plurality of objectsbeing drawn according to the user manipulation, performing a firstcontrol operation corresponding to the first object and a second controloperation corresponding to the second object in association with eachother.

Herein, the performing may include displaying a first screen byexecuting a first program corresponding to the first object, andexecuting, in response to a separate user manipulation being performedon the first screen, a second program corresponding to the second objectand displaying a second screen which a result according to the separateuser manipulation on the first screen is reflected.

The performing may further include determining a content which isdisplayed on the first screen according to a type of the second program.

According to an aspect of another exemplary embodiment, there isprovided a user terminal device including a display configured todisplay a screen including an object, a detector configured to detect auser manipulation on the screen, and a controller configured to, inresponse to a first area of the object being selected according to theuser manipulation, perform a first control operation related to theobject, and in response to a second area of the object being selected,perform a second control operation related to the object.

According to an aspect of another exemplary embodiment, there isprovided a user terminal device, including a controller configured to,in response to a first user manipulation on a screen displaying a firstobject and a second object being detected, determine whether the firstobject and the second object are selected by the first usermanipulation, and in response to it being determined that the firstobject and the second object are selected, execute a first programcorresponding to the first object and a second program corresponding tothe second object. The first program and the second program are executedseparately or in association with each other according to a type of thefirst control operation and the second control operation.

The controller may be further configured to, in response to the firstprogram and the second program being executed separately, simultaneouslydisplay a first screen corresponding to executing the first program anda second screen corresponding to executing the second program.

The controller may be further configured to, in response to the firstprogram and the second program being executed in association with eachother, display a first screen corresponding to executing the firstprogram and, in response to a second user manipulation being performedon the first screen, display a second screen corresponding a result ofthe second user manipulation.

The controller may be further configured to, in response to the firstprogram and the second program being executed in association with eachother, display a screen corresponding to executing the first program andthe second program simultaneously.

According to an aspect of another exemplary embodiment, there isprovided a controlling method of a user terminal device, includingdetecting a user manipulation on a screen displaying a first object anda second object, determining whether the first object and the secondobject are selected by the user manipulation, and executing, in responseto determining that the first object and the second object are selectedby the user manipulation, a first program corresponding to the firstobject and a program corresponding to the second object. The firstprogram and the second program are performed separately or inassociation with each other according to a type of the first controloperation and the second control operation.

In response to the first program and the second program being executedseparately, the executing may include simultaneously displaying a firstscreen corresponding to executing the first program and a second screencorresponding to executing the second program.

In response to the first program and the second program being executedin association with each other, the executing may include displaying afirst screen corresponding to executing the first program, anddisplaying, in response to a second user manipulation being performed onthe first screen, a second screen corresponding a result of the seconduser manipulation.

In response to the first program and the second program being executedin association with each other, the executing may include displaying ascreen corresponding to executing the first program and the secondprogram simultaneously.

According to various exemplary embodiments, a user terminal device mayperform at least one control operation promptly according to a usermanipulation regarding an object. Thus, a user may conveniently receivea service that he/she wants.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will become more apparent by describingcertain exemplary embodiments with reference to the accompanyingdrawings, in which:

FIG. 1 is a view illustrating a configuration of a user terminal deviceaccording to an exemplary embodiment;

FIG. 2 is a flowchart provided to explain a control method of a userterminal device according to an exemplary embodiment;

FIGS. 3 to 8 are views provided to explain examples of various usermanipulations and related operations;

FIG. 9 is a flowchart provided to explain a controlling method of a userterminal device according to another exemplary embodiment;

FIGS. 10 to 12 are views provided to explain an operation of a userterminal device according to the exemplary embodiment of FIG. 9;

FIG. 13 is a flowchart provided to explain a controlling method of auser terminal device according to another exemplary embodiment;

FIGS. 14 to 15 are views provided to explain an operation of a userterminal device according to an exemplary embodiment of FIG. 13;

FIG. 16 is a flowchart provided to explain a controlling method of auser terminal device according to another exemplary embodiment;

FIGS. 17 to 21 are views provided to explain examples of variousoperations of a user terminal device according to the exemplaryembodiment of FIG. 16;

FIG. 22 is a flowchart provided to explain a controlling method of auser terminal device according to another exemplary embodiment;

FIGS. 23 to 27 are views provided to explain examples of variousoperations of a user terminal device according to the exemplaryembodiment of FIG. 22;

FIG. 28 is a view illustrating an external configuration of a userterminal device with a pen;

FIG. 29 is a view illustrating an example of a detailed configuration ofa user terminal device;

FIG. 30 is a view illustrating an example of software structure used bya user terminal device;

FIG. 31 is a view provided to explain an example of a detailedconfiguration of a user terminal device;

FIG. 32 is a view illustrating an example of a configuration of a usercreative screen which is generated by a user;

FIG. 33 is a view illustrating another example of a configuration of auser creative screen which is generated by a user;

FIG. 34 is a view illustrating an example of an authoring tool screen toconfigure a user creative screen;

FIGS. 35 to 37 are views provided to explain a method of configuring auser creative screen according to various exemplary embodiments;

FIGS. 38 to 42 are views provided to explain examples of various methodsof generating a user creative screen; and

FIGS. 43 to 44 are views provided to explain various methods ofutilizing a user creative screen.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Certain exemplary embodiments will now be described in greater detailwith reference to the accompanying drawings.

It should be observed that the method steps and system components havebeen represented by conventional symbols in the figures. In thefollowing description, the same drawing reference numerals are used forthe same elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exemplaryembodiments. Thus, it is apparent that the exemplary embodiments can becarried out without those specifically defined matters. Also, well-knownfunctions or constructions are not described in detail since they wouldobscure the exemplary embodiments with unnecessary detail. Further, inthe following description, relational terms such as first and second,and the like, may be used to distinguish one entity from another entity,without necessarily implying any actual relationship or order betweensuch entities.

FIG. 1 is a block diagram provided to explain a configuration of a userterminal device according to various exemplary embodiments. According toFIG. 1, a user terminal device 100 includes a display 110, a detector120, and a controller 130.

The user terminal device 100 refers to various types of electronicapparatuses which are usable by a user. For example, the user terminaldevice 100 may be realized as various types of devices such as mobilephones, tablet PCs, laptop PCs, personal digital assistants (PDAs), MP3players, electronic album devices, televisions (TVs), PCs, kiosks, andso on. In FIG. 1, only certain elements which are involved in theoperations according to the various exemplary embodiments areillustrated, and illustrations of the remaining detailed elements areomitted.

The display 110 may display various types of screens such as a homescreen, an icon screen, a list screen, an application execution screen,a web browser screen, a content play screen, and so on. The home screenrefers to a screen which is displayed after the user terminal device 100is turned on and preparation for system is completed. The home screenmay also refer to a screen from which an application or a widget can beselected. The home screen may also be referred to as a basic screen, aninitial screen, a main screen, an unlock screen, etc. The icon screenrefers to a screen where icons regarding the applications installed inthe display 110 are displayed, and a list screen refers to a screenwhere information regarding the applications installed in the display110 is arranged and displayed in the form of list. The applicationexecution screen refers to a screen which is rendered by the executionof an application, and a web browser screen refers to a screen where aweb page received from an external web server is displayed by theexecution of a web browser. The content play screen refers to a screenwhere multimedia contents are reproduced and their video frames aredisplayed. In addition, the display 110 may display various other typesof screens.

Various types of objects such as icons, text, images, photos, etc. maybe included on the screen which is displayed on the display 110. Forexample, the home screen may include various objects such as a specificicon selected by a user, various widget information, a folder icon, etc.The detector 120 may detect a user manipulation with respect to thedisplay 110.

A user manipulation may be input through various means such as a user'sfinger or other body parts, a pen (e.g., a stylus), etc., and dependingon the manipulation method, may be classified as a touch manipulation, ahovering manipulation, a gazing manipulation, a motion gesturemanipulation, a voice input manipulation, and so on.

The touch manipulation refers to an operation where a user touches ascreen using an input means, and the hovering manipulation refers to anoperation of approaching an input means close to a screen. The touchmanipulation may include a user's drawing input where a user draws anumber, text, a symbol, a sign, etc. on the screen. The user's drawinginput may be realized as a touch-and-drag manipulation using a user'sbody part or a pen. Meanwhile, the gazing manipulation refers to anoperation where a user gazes part of the screen for a predeterminedtime. The motion gesture manipulation refers to an operation where auser makes a predetermined motion gesture in front of the user terminaldevice 100 so that a control operation matching with the motion gesturecan be performed. The voice input manipulation refers to an operationwhere a user utters a voice command so that a control operation matchingwith the voice command can be performed.

The detector 120 is an element to detect a user manipulation. Thedetector may be realized in various ways depending on a user'smanipulation means and methods. For example, to detect a touchmanipulation, the detector 120 may be realized as various types of touchsensors such as capacitance-type, resistive-type, and piezoelectric-typesensors. The capacitive sensor calculates touch coordinates by sensingelectricity excited by a user's body when part of the user's bodytouches the surface of the display 110 by using a dielectric coated onthe surface of the display 110. The resistive sensor comprises twoelectrode plates, and calculates touch coordinates as the upper andlower plates of the touched point contact with each other to senseflowing electric current when a user touches a screen.

Meanwhile, if an input means is a pen including a coil, the detector 120may include a geomagnetic sensor to sense a magnetic field which ischanged by the coil in the pen. Accordingly, in addition to sensing atouch manipulation, an approach manipulation, that is, a hovering, mayalso be detected. For example, the detector 120 may detect a user'sdrawing input which is performed in an object included in the screen ofthe display 110 or in the vicinity of the object. The configuration ofthe detector 120 will be described in greater detail later.

When a user manipulation is detected, the detector 120 communicates thedetection result to the controller 130. For example, if a user touches acertain point on the screen using user's finger or a pen, the detectorcommunicates x, y coordinates of the touched point to the controller130. When the user moves the touched point, the detector 120communicates the changed coordinates of the touched point to thecontroller 130 in real time. When a user manipulation with respect to anobject displayed on the display 110 is detected, the controller 130 mayperform an appropriate control operation according to the object and theuser manipulation. At least one control operation may be matched witheach object displayed on the screen as a default or by a user's setting.

The controller 130 may perform a control operation corresponding to anobject selected by a user. The control operation refers to executing aspecific application of the user terminal device 100 and displaying itsexecution screen, or performing a predetermined specific function. Inaddition, when a user's drawing input is performed on an object or inthe vicinity of the object, the controller 130 may execute a programcorresponding to the object, and display a screen where the user'sdrawing input is processed as an input of the program on the display110. Hereinafter, the examples of performing various control operationsaccording to an object and a user manipulation will be described ingreater detail.

<Example of Designating Objects in a Lump>

FIG. 2 is a flowchart provided to explain a control operation accordingto an exemplary embodiment. According to FIG. 2, the user terminaldevice 100 may display a screen including a plurality of objects (S210).Herein, the screen may be realized as various screens such as a homescreen, an icon screen, a list screen, an application execution screen,a web browser screen, a content play screen, etc. as described above.For example, the user terminal device 100 may display various icons,widget information, etc. on the home screen. In this state, the userterminal device 100 determines whether a user manipulation ofassociating a plurality of objects is detected (S220).

The user manipulation of associating a plurality of objects may beperformed in various ways. For example, the manipulation may be at leastone of drawing a closed curve including a plurality of objects, drawinga connection line between a plurality of objects, performing amulti-touch manipulation of touching each of a plurality of objects,covering an area where a plurality of objects is displayed, and so on.These manipulations are examples of touch manipulations. A voicemanipulation may include a user manipulation of uttering the titles of aplurality of objects in a row or a user manipulation of uttering a voicecommand which is set to associate a plurality of control operations.

In addition, when a specific motion gesture is detected during a motionmanipulation, it may be determined that the specific motion gesture is auser manipulation of associating a plurality of objects. When such auser manipulation is detected, the controller 130 may perform a controloperation corresponding to each of a plurality of objects which areassociated by the user manipulation (S230). The control operation may beperformed in various ways.

Hereinafter, the examples of a user manipulation of associating aplurality of objects and the corresponding control operation will bedescribed in greater detail.

FIG. 3 is a view illustrating a case where a plurality of objects areselected simultaneously according to a user manipulation of drawing aclosed curve. FIG. 3 illustrates an example where a home screenincluding widget information 11 and various icons 12˜17 is displayed. Inthis state, if a user draws a closed curve 10 on the screen using aninput means such as his or her finger 50 or a pen, the detector 120communicates the touched point to the controller 130. The controller 130recognizes a drawing trace by connecting touched points successively.Subsequently, the controller 130 renders a graphic line 10 inpredetermined color and thickness along the drawing trace on the display110. The graphic line 10 is displayed in an overlapping manner on thehome screen. When there is an overlapping part while the graphic line 10is being drawn, the controller 130 recognizes its internal area as aclosed curve. The controller 130 determines the types of objectsdisplayed within the closed curve.

FIG. 3 illustrates an example where the widget information 11 and theicon B 13 are included in the closed curve consisting of the graphicline 10. In this case, the controller 130 executes a control operationcorresponding to the widget information 11 and the icon B 13,respectively, and displays their execution screens 21, 22 in a row. Thatis, with only one user manipulation, a plurality of applications may beexecuted and accordingly, a plurality of application execution screensmay be displayed.

FIG. 3 illustrates a closed curve where the starting point and thefinishing point of a user manipulation are the same. However, even if auser manipulation does not form a closed curve, the same operation asFIG. 3 may be performed as long as the user manipulation can beinterpreted as an operation of selecting a plurality of objectssimultaneously. For example, if a user's drawing trace forms a curve,but the curve is not closed, the controller 130 checks the entire lengthof the drawing and the distance between traces. If the distance betweentraces are small enough in comparison with the entire length of thedrawing and there is at least one object displayed inside the circle,the controller 130 may determine that a user draws a closed curve.Accordingly, it may be determined that the object inside the circle isselected. According to another example, a plurality of objects may beassociated according to manipulations of drawing various lines otherthan a closed curve. For example, if a user draws a line which crossesobjects A, B, C (12, 13, 14) in turn, the controller 130 may determinethat the objects A, B, C (12, 13, 14) are selected, respectively, andmay associate the corresponding control operations A, B, C to eachobject in a parallel manner and perform the operations.

For example, if control operations are performed in a parallel manner,and a straight line or a curve is drawn on a music file icon, a photoicon, and a message service icon, a music file play screen, a photoscreen, and a text message screen may be lined up and displayed on thefull screen. On the other hand, if control operations are performed inassociation with each other, a control operation, where a text messagescreen, where a music file and a photo are attached, is displayed on thescreen, may be performed. As described above, a plurality of controloperations may be performed separately or in association with each otheraccording to a user manipulation.

FIG. 3 illustrates that when a plurality of objects are grouped or aline connecting a plurality of objects is drawn, a control operationmatched with each object is executed, respectively, but there may beother exemplary embodiments. For example, when a plurality of objectsare grouped as one group, or a plurality of objects are connected witheach other, an editing operation may be performed with respect to theobjects, that is, the objects may be moved, copied, or deleted in alump. Specifically, if one of objects which belong to one group or whichare connected with each other is touched and dragged to another locationby a user, the other objects which are grouped together or connectedeach other may also be moved. In another example, a plurality of objectswhich are grouped together or connected each other may be moved into asingle folder. These operations may also be applied to other exemplaryembodiments which will be described later.

FIG. 4 illustrates a case where a plurality of objects are selected by acovering manipulation of covering an area where the plurality of objectsare displayed at once. As illustrated in FIG. 4, when a user covers aplurality of objects 11, 13, 14 using the entire palm 50 or at least onefinger, the detector 120 provides the controller 130 with informationregarding the covered area. The controller 130 confirms objectsdisplayed in the corresponding area and performs control operationscorresponding to the objects, respectively.

FIG. 4 illustrates an example where widget 11 and two icons 13, 14 arecovered by a user's hand and accordingly, the execution screens 21, 22,23 of programs corresponding to the each of the widget 11 and icons 13,14, respectively, are displayed in order. The detector 120 may detect acovering manipulation using a touch sensor. If a touch area detected bythe detector 120 is larger than a predetermined size, the controller 130may determine that all of the objects displayed in the touched area areselected. Accordingly, as illustrated in FIG. 4, control operationsmatched with each object are performed in a lump.

FIG. 5 illustrates a case where a plurality of objects are selected by amulti-touch manipulation of touching the plurality of objectssimultaneously. If such a multi-touch manipulation is detected, thedetector 120 provides the controller 130 with information regarding thenumber and location of the touched points. If there are a plurality oftouched points and an object is displayed at each of the touched points,the controller 130 determines that the objects are selectedsimultaneously. FIG. 5 illustrates a case where two objects 16, 17 areselected simultaneously. Accordingly, execution screens 31, 32 ofapplications corresponding to each of the selected objects 16, 17,respectively, may be displayed in order.

In FIGS. 3 to 5, the length of the display 110 is greater than the widthof the display 110, but the length and the width of the display 110 maybe realized in various ways depending on the type of the user terminaldevice 100. In addition, if a plurality of applications are executed,the arrangement order of their execution screens may be realized invarious ways depending on the shape and size of the display 110. Thatis, unlike the examples illustrated in FIGS. 3 to 5, in the case where auser terminal device 100 has a display where the width is greater thanthe length, each execution screen may be arranged and displayed in ahorizontal direction.

In FIGS. 3 to 5, each of the applications corresponding to a pluralityof objects is executed and a plurality of execution screens areprovided. However, the controller 130 may also associate controloperations corresponding to each of a plurality of objects which areselected simultaneously by a user manipulation with each other andperform the control operations.

The association operation may be performed in various ways. For example,if a thumbnail photo and a mail icon are selected simultaneously by adrawing manipulation, a covering manipulation, multi-touch manipulation,etc., the controller 130 may generate a mail writing screen where thethumbnail photo is attached automatically and display the screen on thedisplay 110. Accordingly, the process of the related art of executing amail program first, and then searching for a photo to be attached may beavoided, and inputting an attachment command is may omitted, and a usermay directly perform a desired control operation. Similarly, if a mailicon and a schedule icon are selected together, the controller 130 maydirectly generate a schedule generating screen where various scheduleinformation recorded in exchanged mails is reflected. Further, if asearch program icon and a specific content icon are selected together,the controller 130 may automatically search information regarding thecorresponding content and directly display a screen which shows thesearch result.

FIG. 6 is a view provided to explain another manipulation method ofassociating a plurality of objects and a method of performing theassociation operation thereof. As illustrated in FIG. 6, a user may drawa connection line 60 between a plurality of objects.

If the connection line 60 is drawn between the first and the secondobjects 15, 16 from among a plurality of objects, thereby connecting twoobjects 15, 16, the controller 130 of the user terminal device 100 mayassociate the first control operation corresponding to the first object15 with the second control operation corresponding to the second object16, and perform the operations. Specifically, the controller 130 mayexecute the first program corresponding to the first object 15, anddisplay the first screen 41. A user may input a certain usermanipulation with respect to the first screen 41. The controller 130performs an operation according to the user manipulation and as aresult, obtains data of execution result of the first program.

In addition, the controller 130 executes the second programcorresponding to the second object 16. The controller 130 provides thesecond program with the data of execution result which is obtained bythe user manipulation with respect to the first program, and displaysthe second screen which reflects the result. FIG. 6 illustrates anexample where the first object 15 is a camera icon, and the secondobject 16 is a message service icon.

As illustrated in FIG. 6, a user may input a manipulation of drawing theconnection line 60 between the camera icon 15 and the message serviceicon 16 using an input means such as his or her finger 50 or a pen 200.In this case, the controller 130 may execute a camera application, anddisplay a live-view screen 41 first by activating a camera (not shown).In this state, if the user inputs a photographing command, thecontroller 130 directly displays a text message writing screen 42 wherea photographed image 43 is attached automatically.

In FIG. 6, the connection line 60 is illustrated as an arrow, but theshape of the connection line 60 is not necessarily limited thereto. Thatis, the connection line 60 may be realized in various ways, such as asolid line, a dotted line, a curved line, etc. If the connection line 60is realized in an arrow shape as illustrated in FIG. 6, the controller130 may determine the execution order of applications in considerationof the directivity of the arrow. That is, if a predetermined timeelapses after a user's drawing is completed, the controller 130 mayanalyze the trace of the user's drawing and determine the shape. If itis determined that the shape of the drawing has a similarity, which ishigher than a certain ratio, with an arrow pattern registered inpredetermined database, the controller 130 may determine that an arrowis drawn, and may decide a directivity according to the location of thearrowhead. Therefore, the controller 130 execute applications matchedwith each object 15, 16 sequentially based on the direction of the arrowand the location of each object 15, 16.

If an arrow is drawn from the camera icon 15 towards the message icon 16as illustrated in FIG. 6, the controller 130 displays the photographingscreen 41 by executing a camera program first. In this state, if a usercompletes photographing, the controller 130 displays the text messagewriting screen 42 by executing a mail program. On the other hand, if anarrow is drawn from the message icon 16 towards the camera icon 15, thecontroller 130 may display the text message writing screen 42 first. Inthis state, if a writing is completed, the controller 130 displays thephotographing screen 41. If photographing is completed through thephotographing screen 41, the controller 130 may attach the photographedimage automatically and transmit the text message. Unlike an arrow, ifthe connection line has no directivity, the controller 130 may determinethe execution order of applications by recognizing the direction inwhich a connection line is drawn. That is, if a connection line is drawnfrom left to right, a control operation corresponding to a left objectmay be performed first, and if a connection line is drawn from right toleft, a control operation corresponding to a right object may beperformed first.

Meanwhile, even if the manipulation of drawing a connection line isperformed as illustrated in FIG. 6, it may be necessary to perform anassociated control operation. That is, depending on exemplaryembodiments, even if a connection line is drawn as illustrated in FIG.6, each application may need to be executed in parallel, and a pluralityof execution screens may be displayed simultaneously.

In addition, when a plurality of objects are associated with each otherby a connection line, the controller 130 may display each screen inconsideration of the type of two programs which are associated with eachother. That is, when displaying the first screen which is the executionscreen of the former program, the controller 130 may determine thecontents to be displayed on the first screen differently by consideringthe type of the latter program. Herein, the former program refers to thefirst program corresponding to the first object 15 which is close to apoint that is touched first to draw the connection line 60, and thelatter program refers to the second program corresponding to the secondobject 16 which is close to a point that is touched last to draw theconnection line 60.

FIGS. 7 and 8 are views provided to explain operations of the userterminal device 100 according to the above exemplary embodiments. FIG. 7illustrates a state where the connection line 60 is drawn between thefirst object 21 matched with a gallery program and the second object 22matched with an editing program. In this case, the controller 130 maydisplay pre-stored photos and video contents 1˜9 on the first screen 41.In this state, if a user selects one content, the controller 130 maydisplay an editing screen to edit the selected content.

FIG. 8 illustrates a state where the connection line 60 is drawn betweenthe first object 21 matched with a gallery program and the second object23 matched with a message service program. In this case, the controller130 selectively displays only those contents 1, 2, 5, 8 which are set beshared from among the pre-stored contents 1˜9 on the first screen 41. Inaddition, the first screen 41 may also display the menu 31 to view theentire contents.

As such, when a plurality of objects are connected by a connection line,the contents displayed on the execution screen of the former program maybe changed depending on the type and characteristics of the latterprogram.

In addition, the controller 130 may directly display an execution screenof a latter program corresponding to a user manipulation rather thanalways displaying an initial screen. In the above exemplary embodiments,the method of performing a plurality of control operations according toa user manipulation of associating a plurality of objects is describedin detail.

In the above exemplary embodiments, each object is displayed in theshape and size which are designed by the manufacturing company or theapplication writer of a user terminal device, but objects may begenerated as a user of the user terminal device directly draws them.That is, according to another exemplary embodiment, a user may directlygenerate an object on the home screen and match the object with adesired control operation to configure the home screen.

Hereinafter, for convenience of explanation, a home screen which can bedesigned directly by a user will be referred to as a User CreativeScreen (UCS) and a screen which is designed by the manufacturing companyor the application writer of a user terminal device will be referred toas a basic screen. The various exemplary embodiments may be performednot only on the basic screen but also on the user creative screen.

Accordingly to another exemplary embodiment, a user may match aplurality of control operations with one object in advance, and selectthe object to execute the plurality of control operations at once. Inthis case, a user may simply select and use a plurality of controloperations without associating a plurality of objects each time, whichwill be explained in greater detail below.

<Example of Matching a Control Operation with an Object>

FIG. 9 is a flowchart provided to explain a method of controlling a userterminal device 100 according to another exemplary embodiment.

According to FIG. 9, a user terminal device displays 100 at least oneobject (S710). The object is not limited to an icon, and the object maybe displayed in various forms such as text, an image, a photo, etc.

In this state, when a user selects one object (S720), the controller 130of the user terminal device 100 determines whether a plurality ofcontrol operations are matched with the selected object (S730). To doso, a storage (not shown) of the user terminal device 100 may storeinformation regarding control operations matched with each object.

If it is determined that a plurality of control operations are matched,the controller 130 performs each control operation in parallel (S740).For example, if a mail application and a diary application are matchedwith one object and the object is selected, the controller 130 executesthe mail application and the diary application respectively, anddisplays the execution screens of each application simultaneously.

On the other hand, if only one control operation is matched with theselected object, the controller 130 performs the matched controloperation (S750).

FIG. 10 is a view provided to explain an operation of a user terminaldevice 100 according to the controlling method of FIG. 9. According toFIG. 10, the user terminal device 100 displays at least one object 700.In FIG. 10, the object 700 is generated as a user directly draws on thescreen using his or her finger or a pen.

In FIG. 10, if it is assumed that a gallery program and a picture boardprogram are matched with the object 700, when the object 700 isselected, the controller 130 displays the execution screens 51, 52 ofthe gallery program and the picture board program on the screen,respectively. The execution screen 51 of the gallery program may displaythumbnail images of various photos stored in the user terminal device100, and an enlarged photo of one of the various photos. The executionscreen 52 of the picture board program may display an area where a usermay draw a picture, and a picture may be drawn as the finger of the useror a pen moves.

In FIG. 10, a user may draw a picture on the execution screen 52 of thepicture board program while watching the execution screen 51 of thegallery program.

Meanwhile, the number and type of control operations which can bematched with one object may vary. For example, if a video play programrather than the picture board program is matched in FIG. 10, a user mayview photos which have been previously photographed on one screen 51 andwatch a video on the other screen 52.

FIG. 11 is a view illustrating another example of an object where aplurality of control operations are matched. According to FIG. 11, anobject 18 including icons D, E which are matched with different controloperations and an arrow connecting them may be displayed. The shape ofthe object 18 may be drawn directly by a user, or may be provided by themanufacturer or the application developer of the user terminal device100. When the object 18 is selected, the controller 130 performs thecontrol operation corresponding to icon D and the control operationcorresponding to icon E sequentially.

For example, if icon D is matched with a camera program and icon E ismatched with a mail program, when the icon D is selected, the controller130 executes the camera program first to display the photographingscreen. In this state, if a user completes photographing, the controller130 executes the mail program to execute the mail writing screen.

FIG. 12 is a view illustrating another example of an object a pluralityof control operations are matched. According to FIG. 12, an additionalobject 19 which connects the objects may be displayed. When icon D 15 isselected, the controller 130 performs the first control operationcorresponding to the selected icon D 15, and when icon E 16 is selected,the controller 130 performs the second control operation correspondingto the selected icon E 16. On the other hand, when the additional object19 is selected, the first and the second operations are performedsequentially.

In the above exemplary embodiment, when a plurality of controloperations are matched with one object and the object is selected, theplurality of control operations are executed simultaneously orsequentially. However, according to another exemplary embodiment, only apart of a plurality of control operations may be performed according toa manipulation pattern of a user.

<Example of Performing a Different Control Operation According to aManipulation Pattern>

FIG. 13 is a block diagram illustrating a configuration of a userterminal device 100 according to another exemplary embodiment. Accordingto FIG. 13, when a user manipulation is detected (S920) while an objectis displayed (S910), the user terminal device 100 determines its pattern(S930).

That is, according to an exemplary embodiment, the user terminal device100 may store a plurality of manipulation patterns which are set for oneobject and control operations matched with the manipulation patterns.The manipulation patterns may be classified according to the direction,intensity, frequency, manipulation means, etc. of a user manipulation.

For example, if the direction of a user manipulation is considered as astandard, the first user manipulation of crossing one object in ahorizontal direction and the second user manipulation of crossing oneobject in a vertical direction may be regarded as two different patternsof manipulations. Accordingly, the first control operation may bematched with the first user manipulation, and the second controloperation may be matched with the second user manipulation.

Alternatively, if the intensity of a user manipulation is considered asa standard, the first control operation may be matched with a usermanipulation of which intensity, pressure, or touched size is greaterthan a threshold value, and the second control operation may be matchedwith a user manipulation of which intensity, pressure, or touched sizeis lower than a threshold value.

In addition, if the frequency of a user manipulation is considered as astandard, the first control operation may be matched with a single tap,and the second control operation may be matched with a double tap.

In addition, if the manipulation means is considered as a standard, thefirst control operation may be matched with a touch manipulation ofusing a finger, and the second control operation may be matched with atouch manipulation using a pen.

Alternatively, with respect to the first user manipulation of crossingan object in a horizontal direction, a vertical direction, or a diagonaldirection, the first control operation of executing an applicationcorresponding to the object a plurality of times and displaying aplurality of execution screens in order may be matched. On the otherhand, with respect to the second user manipulation (that is, a simpletouch, etc.) of not crossing the corresponding object, the secondcontrol operation of executing an application corresponding to theobject only once and displaying one execution screen on the display 110may be matched.

When the pattern of a user manipulation is confirmed, the controller 130of the user terminal device 100 performs a control operation accordingto the confirmed pattern based on pre-stored matching information(S940). Subsequently, even if the same object is selected, a differentprogram may be executed or a different function may be performedaccording to the selection method. In addition, even if only one usermanipulation is performed with respect to an object, a plurality ofexecution screens may be displayed.

FIGS. 14 and 15 are views illustrating an example of a detailedoperation of a user terminal device 100 according to an exemplaryembodiment. In the above drawings, various exemplary embodiments aredescribed based on the user terminal device 100 where the length of thedisplay 110 is greater than the width of the display 110, but in FIGS.14 and 15, the user terminal device 100 where the width of the displayis greater than the length, such as a tablet PC, notebook PC, etc. willbe described. However, exemplary embodiments are not limited to theabove types of devices, and a user terminal device 100 may be realizedin various forms.

According to FIG. 14, the user terminal device 100 may display aplurality of objects 12˜20. In this state, if a user performs a usermanipulation of crossing one object 19 in a vertical direction, thecontroller 130 executes an application corresponding to the object 19twice and displays two execution screens 61, 62. In this case, thearrangement order of the execution screens 61, 62 may be determinedaccording to the direction of the user manipulation. That is, if a userperforms a user manipulation of crossing one object 19 in a verticaldirection as illustrated in FIG. 14, the execution screens 61, 62 arearranged in left and right sides, respectively, with reference to thescreen.

FIG. 15 illustrates a case where a user crosses one object 19 in ahorizontal direction. In this case, the controller 130 arranges twoexecution screens 61, 62 in upper and low sides, respectively, withreference to the screen.

Meanwhile, when an object is selected, the controller 130 may displaytwo of the same execution screens 61, 62, or may display the twodifferent execution screens 61, 62 according to a predeterminedstandard. For example, as illustrated in FIGS. 14 and 15, if a usermanipulation is performed with respect to the object 19 where a messageservice icon is matched, the controller 130 may execute the messageservice icon, display a message exchange history on one execution screen61 and the last messages exchanged on the other execution screen 62.

In FIGS. 14 and 15, a message service icon is shown as an example, butprograms other than a message service icon may be matched as well.

Meanwhile, a user manipulation may be classified according to amanipulation location. Hereinafter, the example of performing adifferent control operation according to the location of a usermanipulation will be described.

<Example of Performing a Different Control Operation According to aManipulation Location>

FIG. 16 is a flowchart provided to explain an example of performing adifferent control operation according to a manipulation location in oneobject.

According to FIG. 16, if a user manipulation is detected (S1220) whilean object is displayed (S1210), a user terminal device 100 executes anapplication corresponding to the object (S1230). Accordingly, if it isdetermined that the first point in the object is selected (S1240), thefirst control operation corresponding to the point is performed (S1250).On the other hand, if it is determined that another point, that is, notthe first point, is selected, the second control operation is performed(S1260). In FIG. 16, one object is divided into two points and twodifferent control operations are matched with each point. However, theremay be more than two points depending on exemplary embodiments.

FIG. 17 illustrates a state in which an object where a cameraapplication is matched is displayed. In FIG. 17, an object 1300 isdivided into a plurality of parts 1310, 1320. For convenience ofexplanation, (+) area 1310 displayed on the upper right of the object1300 is referred to as the first part, and the remaining area isreferred to as the second part.

As a user touches the first part 1310 and the second part 1320separately, the user may execute a desired detailed function directly.That is, if the first part 1310 is touched by a user manipulation 1311in FIG. 17, the controller 130 executes the camera application andoperates in a panorama photographing mode immediately. Accordingly, ascreen 71 for panorama photographing is displayed.

On the other hand, if the second part 1320 is touched by a usermanipulation 1321, the controller 130 executes the camera application,and a screen 72 according to a general photographing mode is displayed.

According to an exemplary embodiment, a user may select a desiredcontrol operation directly on the screen where the object 1300 isdisplayed without executing an application, searching a menu one by one,and setting a detailed option, which causes inconvenience.

In particular, if a user uses a pen as an input means, it is possible toinput a fine touch as the end part of the pen is sharper than a finger.Accordingly, even though the size of the object 1300 is not big enough,a user may touch a detailed point easily using a pen, thereby executinga desired function immediately.

FIG. 18 is a view provided to explain another example of performing adifferent control operation according to a location of manipulation inone object. According to FIG. 18, a user may perform a hoveringmanipulation of approaching the pen 200 to an object 1810 displayed onthe screen. When the hovering manipulation is detected, the controller130 displays a graphic effect of peeling off one layer of the surface ofthe object 1810 to expose a hidden part 1811 of the object 1810.Accordingly, one object 1810 may be divided into the hidden part 1811and an original part 1812. In this state, the controller 130 performs adifferent control operation according to a part touched by the user.

If the original part 1812 is selected in FIG. 18, the controller 130displays a main execution screen 1813 of the corresponding object 1810,and if the hidden part 1811 is selected, the controller 130 displays asimple pop-up screen 1814 related to the corresponding object 1810.However, such a control operation is only an example, and variouscontrol operations may be performed according to the type of programs.

In FIG. 17, it is explained that one object is divided into two parts,fixedly and in FIG. 18, one object is divided into two parts flexiblyaccording to a hovering manipulation. However, according to anotherexemplary embodiment, the shape of object may change depending on changein circumstances of the user terminal device 100, or an additionalobject may be added to one side of the object.

FIGS. 19 and 20 are views provided to explain examples where anadditional object is added.

FIG. 19 illustrates a state in which an object 1822 where a messageservice program is matched is displayed. In this state, if a new textmessage is received but not opened, the controller 130 displays anadditional object 1821 on one side of the object 1822. The additionalobject 1821 may be displayed as the number of new text messages, orother texts, symbols, images, etc. A user may select the original object1822 or the additional object 1821 using various input means. Thecontroller 130 performs a different control operation according to thepart selected by the user.

For example, if a user selects the additional object 1821, thecontroller 130 displays a message screen 1823 for opening a new textmessage immediately. On the other hand, if a user selects the originalobject 1822, the controller 130 displays an execution screen 1824 of amessage service program. In FIG. 19, the message service program istaken as an example, but the same exemplary embodiment can be applied toa messenger program or a mail program.

FIG. 20 illustrates a state in which an object where a certain programis matched is displayed. A user may register his or her scheduleinformation any time. If it is determined that a user's registeredschedule is approaching, the controller 130 displays the additionalobject 1831 to inform the advent of the new schedule on one side of theobject 1832. The user may select the additional object 1831 or theoriginal object 1832.

If the user selects the additional object 1831, the controller 130displays a screen 1833 to inform the content of the new schedule. On theother hand, if the user selects the original object 1832, the controller130 displays an execution screen 1834 of a program corresponding to theoriginal object 1832.

As such, a different control operation may be performed according to auser's selection point even with respect to one object.

Meanwhile, the programs of the exemplary embodiments of FIGS. 17 to 20are merely exemplary, and the same exemplary embodiments may be appliedto other types of programs. Specifically, any one of exemplaryembodiments in FIGS. 17 to 20 may be realized in various programs suchas a content play program, a news program, a game program, anaddress-book program, a telephone program, a web browser program and soon.

For example, suppose that an object, where a content play program ismatched in the exemplary embodiment of FIG. 17, is divided into aplurality of parts. In this case, if the first part from among theplurality of parts is selected, the controller 130 may display a playscreen of the previously-played content, and if the second part which isdifferent from the first part is selected, the controller 130 maydisplay a list of playable contents.

Alternatively, suppose that the exemplary embodiment of FIG. 18 or FIG.19 is applied to a news program. In this case, when there is a breakingnews, the controller 130 may display an additional object on one side ofthe object where the news program is matched to inform a user of thebreaking news. If a user selects the additional object, the controller130 may display a news screen which displays the breaking news. On theother hand, if the original object is selected, the controller 130 maydisplay a main screen of the news program.

As described above, even with respect to an object where one program ismatched, various execution screens may be selectively provided accordingto a user manipulation regarding the object.

Meanwhile, in FIGS. 17 to 20, different execution screens are displayedaccording to the selection point of an object, but other types ofcontrol operations, e.g., other than displaying an execution screen, mayalso be performed.

FIG. 21 is a view provided to explain another exemplary embodiment ofperforming various control operations according to a selection point ofan object.

According to FIG. 21, the controller 130 may display a screen includinga circular object 1840 and various icons 1847. The circular object 1840may include various detailed objects 1841˜1846. A user may performvarious control operations by manipulating each of the detailed objects1841˜1846. FIG. 21 illustrates a case where a widget to play a musiccontent is realized as the circular object 1840. The circular object1840 may include various buttons 1841, 1842, 1843 which are selectableby a touch, a volume menu 1844 which is adjustable through atouch-and-drag, a play state adjustment menu 1845, a play state displayarea 1846, etc.

If a user selects various buttons 1841, 1842, 1843, the controller 130may perform operations such as content play, fast forward, rewind, etc.depending on the type of selected button. Alternatively, if a usertouches an area where the volume menu 1844 is displayed using the pen200 and drags it in a clockwise or counterclockwise direction, thecontroller 130 may increase or decrease volume level according to thedragging direction.

If a user touches an area where the play state adjustment menu 1845 isdisplayed using the pen 200 and drags it in a clockwise orcounterclockwise direction, the controller 130 may move a play pointforward or backward according to the dragging direction. The play statedisplay menu 1846 may be realized as a bar graph on the outermost sideof the circular object 1840. If a user adjusts the play state displaymenu 1845, the controller 130 may change the length of the bar graphdisplayed on the play state display area 1846 according to theadjustment state.

If a circular object is provided as illustrated in FIG. 21, a user mayperform various interaction operations by making a gesture of drawing acircle around the border of the circle. The area where the play stateadjustment menu 1845 is displayed in the circular object 1840 of FIG. 21may be displayed in the form of disk where music contents are recorded.That is, a user may perform a deejay operation of grapping andscratching the disk using the pen 200.

In FIG. 21, a circular object is taken as an example, but other types ofvarious objects which can be easily manipulated by the pen 200 may beprovided, and the controller 130 may perform various control operationsaccording to a selection point of the object.

Meanwhile, a user may perform a user manipulation not only to select anobject but also to input specific information. In this case, thecontroller 130 may execute a desired function directly by combining thearea where the user manipulation is performed or the type of objectdisplayed in the surrounding area with the content of the usermanipulation. Herein after, the exemplary embodiment of performing acontrol operation which reflects a user input will be described.

<Example of Performing a Control Operation which Reflects a User Input>

FIG. 22 is a view provided to explain a method of controlling a userterminal device 100 according to another exemplary embodiment. Accordingto FIG. 22, a user terminal device displays 100 various objects such asa widget and an icon (S1410). In this state, a user may write text ordraw a symbol on the object or in the surrounding area using an inputmeans. That is, a user may input various information through a userdrawing input. The user terminal device 100 may perform a differentcontrol operation based on the type of the object where the user drawinginput is performed and the content of the user input.

The user drawing input may include an input to specify an object and aninput of program corresponding to the object. Specifically, if a userdrawing input is performed, the controller 130 determines that an objectdisplayed at a point where the user drawing input starts is selected,and analyzes the trace of the entire user drawing input. If it isdetermined that the analyzed trace is a specific text or symbol, thecontroller 130 provides the determined text or symbol as an inputregarding a program corresponding to the object. Accordingly, theprogram configures a program execution result screen according to thetext or symbol, and the controller 130 displays the execution resultscreen through the display 110.

That is, if a user drawing input is detected on an object or in thesurrounding area, the controller 130 may execute a program correspondingto the object, and processing the user drawing input as an input to thecorresponding program and display the screen on the display 110.

FIG. 22 illustrates a case where a text is input.

When it is detected that a text is input (S1420), a user terminal device100 executes an application corresponding to the object where the textis input (S1430). Subsequently, a control operation is performed basedon the text input by a user (S1440). Specifically, the controller 130provides the text input by the user as an input regarding the executedapplication, and displays the execution result screen.

FIG. 23 is a view provided to explain an operation of a user terminaldevice 100 according to an exemplary embodiment. FIG. 23 illustrates astate where various objects 11˜19 including a widget, an icon, etc. aredisplayed. A user may perform a certain user manipulation with respectto the objects.

The detector 120 detects such user manipulations and communicates themto the controller 130.

The controller 130 performs a control operation corresponding to thecontents of user manipulations and the type of the correspondingobjects. For example, if a text is drawn in an area corresponding to anobject, that is, if a text is drawn on an object or in the surroundingarea by a pen manipulation, the controller 130 executes a programcorresponding to the object and display an execution screencorresponding to the text on the display 110 immediately.

FIG. 23 illustrates a case where a user draws a certain text, forexample, “KIM” on a search widget 11. In this case, the controller 130executes a program corresponding to the search widget 11 and inputs“KIM” as a search word to perform the search. Subsequently, thecontroller 130 displays the search result screen 1400. As a result,various search results which include the search word, “KIM” on thesearch result screen 1400.

In the related art, when the search widget 11 is selected, an initialexecution screen of the application is displayed first and then, a userhas to input a search word on a search window of the initial executionscreen to perform the search. However, according to an exemplaryembodiment, a user may input a search word on the home screen and checkthe search result immediately, so the user may use a service morequickly and conveniently.

FIG. 23 illustrates a case where a text drawing is performed on thesearch window, but even if a text drawing is performed within a certaindistance with reference to the search window, a search service based onthe corresponding text may be provided. Meanwhile, a search service istaken as an example in FIG. 23, but the exemplary embodiment may beapplied to other various programs.

FIG. 24 illustrates a case where a user writes a text 1601, “To KIM”, onone side of the object 1600 where a mail program is matched using afinger or a pen, and draws a circle 1602 around the text. In this case,the controller 130 executes the mail program and parses the contentsdrawn by the user.

The controller 130 separates a command and a search word from the parsedcontents. In FIG. 24, “To” belongs to a command. The controller 130makes comparison to find out whether there is a word corresponding tothe parsed word exists in a pre-stored command library file, and if itexists, recognizes the word as a command. The controller 130 configuresa screen corresponding to the command.

In FIG. 24, the controller 130 determines that the command, “To”, is acommand to write and transmit a mail, and displays a mail writing screen1610 corresponding to the command. FIG. 24 illustrates that a circle1602 is drawn after the text-writing, but the circle 1602 may beomitted. That is, the controller 130 may disregard the circle 1602 andrecognize only the remaining area as a text. Alternatively, if a closedcurve such as the circle 1602 is recognized, the controller 130 maydetermine that the text-writing is completed and parse only the textwithin the closed curve. In addition, in the example of FIG. 24, thecontroller 130 determines the text, “KIM”, as a search word.

Accordingly, the controller 130 search a mail address corresponding tothe search word in pre-stored contact information. To do so, informationregarding commands corresponding to functions provided by eachapplication may be stored a storage (not shown) in advance. When a mailaddress corresponding to a search word input by a user is found, thecontroller 130 directly writes down the mail address in a receiverinformation area in the mail writing screen.

Accordingly, the mail writing screen 1610 including the mail address ofa counterpart input by a user may be directly displayed. Therefore, theuser may send a mail directly to a desired counterpart without goingthrough the complicated process of selecting various menus or searchprocess.

If a text is input on an icon where a messenger program, rather than amail program, is matched or in the surrounding area, the controller 130may execute the messenger service program and display a messenger screenfor displaying contents of messengers which are exchanged with anothercorresponding to the text on the display 110.

FIG. 25 illustrates a case where a text drawing is performed on anobject where a telephone program is matched. According to FIG. 25, if auser draws a text 1621, “KIM”, on an object 1620 where a telephoneprogram is matched, the controller 130 automatically searches atelephone number corresponding to “KIM” in an address book. If thetelephone number is found, the controller 130 automatically makes a callto the telephone number. Accordingly, a telephone call connection screen1622 is displayed.

Meanwhile, in the case where a user directly touches an object such as awidget and an icon and changes a setting value of a programcorresponding to the object, the shape of the object may be changedaccording to the changed value.

FIG. 26 is a view provided to explain an example of changing a widgetaccording to a user manipulation. According to FIG. 26, the display 110displays a screen 2000 including a widget 1630.

FIG. 26 illustrates that the widget 1630 is an alarm widget whichincludes an analog clock 1631 with an hour hand 1634 and a minute hand1635. The alarm widget 1630 may also display a digital clock 1632, dateinformation 1633, etc.

The controller 130 sets an alarm according to a user manipulation on thealarm widget 1630, and changes the display contents of the widgetaccording to the set alarm. Specifically, the controller 130 may displayonly a part of numbers (i.e., 12, 3, 6, 9) indicating the time on theanalog clock 1631 of the widget 1630, and if there is a pen hovering,may display all the numbers (1˜12) indicating the time on the analogclock. In addition, a user may touch each of the hour hand 1634 and theminute hand 1635 and drag them 1641 to a desired location.

If a pen manipulation for moving at least one of the hour hand 1634 andthe minute hand 1635 is input on the analog clock 1631, the controller130 sets an alarm to the time which is changed by the pen manipulation.Subsequently, at least one of the hour hand and the minute hand may bemoved to the time where the alarm is set. In FIG. 26, the hour hand 1634is moved from 11 to 5, and the minute hand 1635 remains at 12.Accordingly, the alarm is set to 5 o'clock. If the time set for alarm ischanged, the time displayed not only on the analog clock 1631 but alsoon the digital clock 1632 is changed automatically.

Meanwhile, a user may touch and select a date where he or she wishes toset an alarm in the date information 1633 using the pen 200. Theselected date may be displayed in a different color from other dates. Inaddition, although not illustrated in FIG. 26, an area for selecting themorning and the afternoon, an area for selecting on/off of alarm, anarea for selecting a bell sound, etc. may also be included in the widget1630. As described above, the pen 200 allows a user to perform a finertouch in comparison with a finger, so the widget 1630 of FIG. 26 may bedisplayed in a small size when the pen 200 is used.

FIG. 27 is a view provided to explain another method of setting an alarmin the alarm widget 1630. According to FIG. 27, a user may write timeinformation 1642 directly on the digital clock 1632 using the pen 200.FIG. 27 illustrates that “1” is written in an hour area, and “27” iswritten in a minute area. The controller 130 recognizes the writing, andsets an alarm time automatically according to the recognition result.Subsequently, the controller 130 changes the display contents of thedigital clock 1632 and the analog clock 1630 according to the set alarmtime.

As described above, a different control operation is performed accordingto a location of pen manipulation not only on an icon but also on awidget, and an input using a pen may also be directly reflected on thewidget program. In addition, various control operations may be performedaccording to the type of program and the contents of a user's writing.

FIGS. 23 and 24 illustrate a case where a user inputs a certain text,but a text or drawing pattern may be set different for each modeprovided by the corresponding application, and an operation may beperformed according to the user manipulation of drawing such a text ordrawing pattern.

For example, in the case of a camera program, there may be variousphotographing modes such as general mode, best mode, face recognitionmode, panorama mode, smile shot mode, video photographing mode, etc. Thestorage 140 may store a text or drawing pattern corresponding to eachphotographing mode. For example, alphabet “B” may set for the best mode,a circle for the face recognition mode, a wave shape for the panoramamode, alphabet “S” for the smile shot mode, and alphabet “M” for thevideo photographing mode.

Accordingly, if a user's drawing is detected on an object where a cameraprogram is matched or in the surrounding area, the controller 130executes the camera program and directly displays a screen according toa photographing mode matched with the drawing manipulation. For example,if it is detected that alphabet M is drawn, a screen for making a videocontent is displayed directly. On the other hand, if there is a simpletouch on the corresponding object, the controller 130 may display abasic screen of the camera program.

In the related art, a user executes the camera program, selects a menuon an initial screen, checks a list of selectable photographing modes,and then, displays a desired photographing mode from the list. However,according to the exemplary embodiment, a desired depth of screen may bedirectly displayed on the home screen, so user convenience may beimproved significantly. In another example, the similar operation may beperformed in an object where a note program for allowing a user to takevarious kinds of notes using a pen is matched. The note program displaysvarious types of papers where a user take notes freely. For example,papers with various layouts such as a note paper with spaces just like ageneral note, an idea paper for writing down various ideas, a letterpaper in the form of letter, a card paper in the form of card, etc. maybe provided.

Each paper may be matched with a different text or drawing pattern andstored in the storage 140. For example, the note paper may be matchedwith a general touch manipulation, alphabet “I” with the idea paper,alphabet “M” with the letter paper, alphabet “C” with the card paper,etc. The controller 130 may display the shape of paper corresponding toa user's drawing pattern immediately based on such matching information.

As illustrated in FIGS. 14 and 15, a specific drawing pattern such as ahorizontal line or a vertical line may be matched with a controloperation to display a plurality of execution screens of a program,regardless of the operation mode or function of the correspondingprogram. In addition, various control operations may be matched with auser's drawing pattern. For example, a user may draw a predeterminedsymbol, text, etc. on an object to edit the object or an applicationmatched with the object.

For example, if a user draws “X” on one object, the controller 130 maydelete the object. Depending on an exemplary embodiment, an applicationmatched with the object may also be deleted. In addition, if a userdraws “C” on one object, the controller 130 may copy the object.

Further, a user may draw “P” to perform a paste operation, and may draw“S” to perform a cut operation. The example of display contents and thecorresponding operations may be realized in various ways. In addition,in the case of a diary object, if a user writes a specific date on thediary object, the controller 130 may execute the diary program and openthe diary of the corresponding date. In addition, in the case of a musicobject, if a user writes a song title, singer name, album name, etc. onthe music object, the controller 130 may search a music contentcorresponding to the writing of the user from pre-stored contents or anexternal server and play the music content immediately.

As such, according to various exemplary embodiments, a user may select adesired control operation using various manipulation methods on thescreen where objects are displayed. Accordingly, user convenience inexecuting and using various programs may be improved significantly.

The above exemplary embodiments may be realized and applied to a deviceindividually, but they may also be combined together and applied. Thatis, a plurality of objects may be grouped in one user terminal device100 and a plurality of control operations may be performedsimultaneously. Alternatively, a plurality of control operations may bematched with one of the objects, and various control operations may beperformed selectively according to the direction or location of the usermanipulation of selecting the object, the contents input by the user,etc. The user manipulation may be realized in various ways such astouch, voice, motion, etc. as described above.

When a user terminal device 100 is controlled by a touch manipulation,the detector 120 may be realized in the form of touch screen. In thiscase, a touch means is not limited to a body part such as a finger, andvarious types of pens may be used. Hereinafter, examples of detailedconfigurations of a user terminal device 100 which can be controlledusing a user's finger or a pen will be described in greater detail.

<Example of a Detailed Configuration of a User Terminal Device which canbe Manipulated>

FIG. 28 is a view illustrating a configuration of a user terminal device100 which can be manipulated using a pen. The pen 200 may be realizedsimply as a touch pen, or as a stylus pen, digitizer pen, etc. In thiscase, a resonant circuit may be formed inside the pen 200. When the pen200 approaches the user terminal device 100 or touches the user terminaldevice 100, an induction magnetic field is generated by a magneticsignal transmitted from a loop coil built inside the user terminaldevice 100. The detector 120 formed in the user terminal device 100detects an induction magnetic field so as to detect the location of anapproaching pen or a touched location.

FIG. 29 illustrates an example of a detailed configuration of the userterminal device 100 which can detect whether a manipulation is input bythe pen 200 or not. According to FIG. 29, the user terminal device 100includes the display 110, the detector 120, the controller 130, and thestorage 140.

The display 110 displays various screens as described above. The display110 may be realized in various forms such as Liquid Crystal Display(LCD), Organic Light Emitting Diodes (OLED) display, Plasma DisplayPanel (PDP), etc. The display 110 may also include a driving circuit, abacklight unit, etc. which can be realized in the form such as a-Sithin-film-transistor (TFT), Low Temperature Poly Silicon (LTPS) TFT,Organic TFT (OTFT), etc.

The display 110 and the detector 120 may be realized as one touchscreen. The detector 120 may include a pen recognition panel 121 and atouch panel 122. The pen recognition panel 121 detects an approach inputor a touch input of the pen 200, and outputs an event signal accordingto the detection result. The pen recognition panel 121 may be realizedin an Electron Magnetic Resonance (EMR) method, and a touch or anapproach input may be detected according to the change in the intensityof magnetic field by an approach or a touch of the pen 200.

Specifically, the pen recognition panel 121 may include an electronicinduction coil sensor (not shown) and an electronic signal processingcircuit (not shown). The electronic induction coil sensor is configuredto have a grid structure where a plurality of loop coils are crossedwith each other. The electronic signal processing circuit provides an ACsignal to each loop coil of the electronic induction coil sensorsequentially, and transmits a signal output from each loop coil to thecontroller 130. When there is the pen 200 including a resonant circuitin the vicinity of a loop coil of the pen recognition panel 121, themagnetic field transmitted from the corresponding loop coil generates anelectric current based the mutual electromagnetic induction in theresonant circuit of the pen 200. Based on the electric current, aninductive magnetic field is generated from a coil constituting theresonant circuit of the pen, and the pen recognition panel 121 detectsthe inductive magnetic field from a loop coil which is in the state ofreceiving a signal so as to detect an approach location or a touchlocation of the pen.

In the above exemplary embodiment, a pen is taken as an example of inputmeans, but any object capable of generating an electric current based onelectromagnetic induction can be used as an input means. The penrecognition panel 121 is disposed at a lower side of the display 110,and may be activated according to a specific event or as a default.

The touch panel 122 is an element to detect a physical touch inputthrough a user's body part or other objects. For example, the touchpanel 122 may be realized in the form of touch film, touch sheet, touchpad, etc. When a touch is detected, the touch panel 122 outputs a touchevent value corresponding to the touched point. As described above, thetouch panel 122 may include various types of touch sensors such ascapacitance-type sensor, resistive-type sensor, etc. For example, whenthe touch panel 122 is realized as a capacitance-type sensor, the touchpanel 122 coat the both sides of glasses with a thin metal conductivematerial (such as Indium Tin Oxide (ITO) layer) so that an electriccurrent may flow on the surface of the glasses, and the touch panel 122is coated and configured as a dielectric substance which may store anelectric charge. When the surface of the touch panel 122 is touched, acertain amount of electric charge is moved to the touched location bystatic electricity, and the touch panel 122 detects the touched locationby recognizing the change in the amount of electric current as a resultof the movement of the electric charge and traces a touch event.

Herein, the touch event which occurs in the touch panel 122 may begenerated mainly by a human finger, but may be generated by any otherobjects such as conductive materials which may cause the variation ofthe amount of electric current. FIG. 29 illustrates that the penrecognition panel 121 and the touch panel 122 are provided as a separatepanel, but the two panels may be realized as one panel. As described inFIG. 29, a user may manipulate the user terminal device 100 using theuser's finger or other body parts, or using the pen 200.

For convenience of explanation, a manipulation using the pen 200 isreferred to as a pen manipulation, and the other manipulations arereferred to as general manipulations. According to the configuration ofFIG. 29, the detector 130 may recognize a pen manipulation and a generalmanipulation distinctively.

In addition, in order to prevent any conflict between a controloperation according to an interaction based on a general manipulationand a control operation according to an interaction based on a penmanipulation, the controller 130 may process the two interactionsdistinctively. Specifically, if a pen manipulation is detected whenpredetermined conditions are satisfied, a control operationcorresponding to the contents drawn by a pen manipulation and an objectdisplayed on the area where the pen manipulation occurs may beperformed. On the other hand, if a pen manipulation is detected whenpredetermined conditions are not satisfied, the controller 130 mayregard the pen manipulation as a general manipulation. That is, eventhough a pen manipulation is performed, a general operation which isdifferent from the above-described control operations may be performedif predetermined conditions are not satisfied.

For example, if a line is drawn between two objects while predeterminedconditions are not satisfied, the controller 130 may not execute controloperations of the two objects in association with each other andinstead, may change or scroll the home screen in left, right, up, ordown direction by page unit according to the direction of the penmanipulation. Alternatively, the controller 130 may just draw a graphicline according to the pen manipulation without performing any particularcontrol operation.

Such conditions may be set in various ways. For example, one conditionmay be performing a drawing by a pen manipulation while a specific menudisplayed on the screen or buttons (volume button, home button, penbutton, hidden key, etc.) on the main body of the user terminal device100 are pressed by a finger. Even if a plurality of objects arespecified by a pen manipulation while the corresponding menu or buttonis not pressed, the controller 130 may simply draw a line correspondingto the pen manipulation without performing any specific operation. Onthe other hand, if a plurality of objects are specified by a penmanipulation while the corresponding menu or button is pressed, thecontroller 130 recognizes the plurality of objects which are associatedby the pen manipulation, and performs a plurality of control operationscorresponding to each object.

Otherwise, one of the conditions may be a state in which the pen 200 isseparated from the main body. In this case, after the pen 200 isdetached, if a user drags the screen using the pen 200, the controller130 may draws a line corresponding to the drawing trace. On the otherhand, if a user drags the screen using a finger while the pen isdetached, the screen may be changed or scrolled by page unit.

In addition, various conditions may be set. The conditions may be set asa default by the manufacturer of the user terminal device 100 and storedin the storage 140, but may also be arbitrarily set by a user while theuser uses the device.

Meanwhile, albeit not illustrated in FIG. 29, the user terminal device100 may further include a driver (not shown) to drive the display 110, aframe buffer (not shown), etc. The frame buffer is an element to storean image frame to be displayed on the panel of the display 110,temporarily, and the driver is an element to convert an image framestored in the frame buffer to an image source signal and provide it tothe display 110 so that the display 110 may display the image frame.

The controller 130 controls overall operations of a user terminal device100 using various programs and data stored in the storage 140. Thecontroller 130 includes an RAM 131, an ROM 132, CPU 133, a graphicprocessing unit (GPU) 134, and a bus 135. The RAM 131, the ROM 132, theCPU 133, and the GPU 134 may be connected to each other through the bus135. The CPU 133 access the storage 140, and performs booting using anoperating system (OS) stored in the storage 140. In addition, the CPU133 performs various operations using various programs, contents, data,etc. stored in the storage 140.

The ROM 132 stores a set of commands for system booting. If a turn-oncommand is input and power is supplied, the CPU 133 copies an OS storedin the storage 140 onto the RAM 131 according to a command stored in theROM 132 and boots a system by executing the OS. If the booting iscompleted, the CPU 133 copies various application programs stored in thestorage 140 onto the RAM 131 and performs the various operations byexecuting the application programs copied in the RAM 131.

If the booting of the user terminal device 100 is completed, the GPU 134displays a home screen. Specifically, the GPU 134 may generate a homescreen including various objects such as an icon, an image, text, etc.using a computing unit (not shown) and a rendering unit (not shown). Thecomputing unit computes property values such as coordinates, a shape, asize, and a color of each object to be displayed according to the layoutof the screen. The rendering unit generates a screen with variouslayouts including objects based on the property values computed by thecomputing unit. The screen generated by the rendering unit is providedto the display 110 and displayed in an display area.

FIG. 30 is a view illustrating an example of software structure used bythe user terminal device 100.

The software of FIG. 30 may be stored in the storage 140, but is notlimited thereto. The software may be stored in various types of storagemeans used in the user terminal device 100.

According to FIG. 30, the user terminal device 100 includes an operatingsystem (OS) 141, a kernel 142, middleware 143, and an application 144.The OS 141 controls and manages overall operations of hardware. That is,the OS 141 is a layer which performs basic functions of hardwaremanagement, memory, and security, etc. The kernel 142 functions as apath to transmit various signals detected by the detector 120, etc. tothe middleware 143. The middleware 143 includes various software modulesto control the operations of the user terminal device 100.

According to FIG. 30, the middleware 143 includes an X11 module 143-1,an APP manager 143-2, a connection manager 143-3, a security module143-4, a system manager 143-5, a multimedia framework 143-6, a UIframework 143-7, a window manager 143-8, and a handwriting recognizingmodule 143-9. The X11 module 143-1 is a module which receives variousevent signals from various hardware in the user terminal device 100.

Herein, the event may be set in various ways, such as an event where auser manipulation is detected, an event where a system alarm occurs, anevent where a specific program is executed or terminated, etc. The APPmanager 143-2 is a module to manage the execution state of variousapplications which are installed in the storage 140. When an event wherean application execution command is input is detected from the X11module 143-2, the APP manager 143-2 calls and executes an applicationcorresponding to the event.

The connection manager 143-3 is a module to support wired or wirelessconnection to a network. The connection manager 143-3 may includevarious specific modules such as a DNET module, UPnP module, etc. Thesecurity module 143-4 is a module to support certification, permission,secure storage, etc. with respect to hardware. The system manager 143-5monitors status of each element of the user terminal device 100 andprovides the result to other modules. For example, when a battery isrunning short, when an error occurs, or when communication disconnectionoccurs, the system manager 143-5 may output an alarm message or an alarmsound by providing the monitoring result to the UI framework 143-7. Themultimedia framework 143-6 is a module which is used to reproducemultimedia contents stored in the user terminal device 100 or which isused to reproduce multimedia contents provided from an external source.The multimedia framework 143-6 may include a player module, a camcordermodule, a sound processing module, etc. Accordingly, an operation ofreproducing various multimedia content, thereby generating andreproducing screens and sounds, may be performed. The UI framework 143-7is a module to provide various UIs. The UI framework 143-7 may includean image composite module to configure various objects, a coordinatescomposite module to calculate the coordinates where an object is to bedisplayed, a rendering module to render the configured object on thecalculated coordinates, a two-dimensional/three-dimensional (2D/3D) UItoolkit to provide a tool for configuring a UI in 2D or 3D form, etc.The window manager 143-8 may detect a touch event by a user's body partor pen or other input events. When such an event is detected, the windowmanager 143-8 transmits an event signal to the UI framework 142-7 sothat an operation corresponding to the event is performed.

The handwriting recognizing module 143-9 is a module to parse andrecognize a trace that a user draws on the surface of the display 110.The handwriting recognizing module 143-9 receives coordinates values ofpoints touched from the pen recognition panel 121 and stores thecoordinates values of the touched points as a stroke. In addition, thehandwriting recognizing module 143-9 may generate a stroke array usingthe stroke. Subsequently, the handwriting recognizing module 143-9 mayrecognize a handwriting by comparing a pre-stored handwriting librarywith the generated stroke array. The recognized handwriting is providedto the application manager 143-22, and the application manager 143-2provides the handwriting to an application corresponding to the objectwhere the user's handwriting occurs. Accordingly, as described above,the application may perform a control operation based on the recognizedhandwriting.

Meanwhile, if a user touches or draws on the surface of the display 110using various input means such as the user's finger or pen, the UIframework 143-7 may draw various graphic lines according to the user'smanipulation. Specifically, if a user draws a picture on the screenusing the pen 200, the controller 130 draws a graphic line on each pointwhere the pen 200 passes through according to the execution of the UIframework 143-7. The thickness of the line may vary according to thepressure of the pen 200 on the screen or the size of the screen touchedby the pen 200. The color and shape of the line may also vary accordingto a user setting. The application module 144 includes applications144-1˜144-n to support various functions. For example, the applicationmodule 144 may include program modules to provide various services suchas a navigation program module, a game module, an electronic bookmodule, a calendar module, an alarm management module, etc. Theseapplications may be set as default, or may be installed and used by auser temporarily.

The software structure illustrated in FIG. 30 is only an example andthus, part of the configuration may be omitted or changed, or newconfiguration may be added, if necessary. For example, the storage 140may further include various programs including messaging modules such asa sensing module to analyze sensed signals in various sensors, amessenger program, Short Message Service (SMS) & Multimedia MessageService (MMS) programs, an e-mail program, etc., a call info aggregatorprogram module, a VoIP module, a web browser module, etc.

Meanwhile, as described above, a user terminal device 100 may berealized as various types of apparatuses such as a mobile phone, atablet PC, a laptop PC, a PDA, an MP3 player, an electronic albumdevice, a TV, a PC, a kiosk, and so on. Herein, when a user terminaldevice 100 is realized as an apparatus with various functions includinga communication function, a content play function, such as a mobilephone, a tablet PC, etc., the user terminal device 100 may furtherinclude elements to perform such functions.

FIG. 31 is a view illustrating a more detailed example of a userterminal device 100 including various elements.

According to FIG. 31, the user terminal device 100 may include thedisplay 110, the detector 120, the controller 130, the storage 140, acommunicator 150, a Global Positioning System (GPS) chip 161, a videoprocessor 162, an audio processor 163, a button 164, a microphone 165, acamera 166, a speaker 167 and a movement detector 168.

The display 110 and the detector 120 may be realized as a touch screenas described above, and may display various screens and detect a usermanipulation on the screen. The controller 130 controls overalloperations of the user terminal device 100 using various programs anddata stored in the storage 140. As the display 110, the detector 120 andthe controller 130 have already been described in the above variousexemplary embodiments, further description will not be provided.

The communicator 150 is an element to perform communication with varioustypes of external apparatuses according to various types ofcommunication methods. The communicator 150 includes a WiFi chip 151, aBluetooth chip 152, a wireless communication chip 153, and an Near FieldCommunication (NFC) chip 154. The WiFi chip 151 and the Bluetooth chip152 may perform communication according to a WiFi™ method and aBluetooth™ method, respectively. When the WiFi chip 151 or the Bluetoothchip 152 is used, various connection information such as SSID, sessionkey, etc. is received/transmitted in advance so that various informationcan be received/transmitted using the same.

The wireless communication chip 153 refers to a chip which performscommunication according to various communication standards such as IEEE,Zigbee, 3rd Generation (4G), 3rd Generation Partnership Project (3GPP),Long Term Evolution (LTE), and so on. The NFC chip refers to a chipwhich operates according to an NFC method by using 13.56 MHz from amongvarious RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz,860˜960 MHz, 2.45 GHz, and so on.

The controller 130 performs communication with various externalapparatuses using the communicator 150. For example, the controller 130may download an application, install the application in the storage 140,and display an object corresponding to the application. In this case,the corresponding application may be matched with a previously-usedobject. The GPS chip 161 is an element to receive a GPS signal from aGPS satellite and calculate the current location of the user terminaldevice 100.

When a navigation program is used or when the current location of a useris required, the controller 130 may calculate the location of the userusing the GPS chip 161. The video processor 162 is an element to processcontents received through the communicator 150 or video contentsincluded in the contents which are stored in the storage 140.

The video processor 162 may perform various image processing withrespect to video data, such as decoding, scaling, noise filtering, framerate conversion, resolution conversion, etc. The audio processor 163 isan element to process contents received through the communicator 150 oraudio data included in the contents stored in the storage 140. The audioprocessor 163 may perform various processing with respect to audio data,such as decoding, amplification, noise filtering, etc. When a playprogram regarding a multimedia content is executed, the controller 130may reproduce the corresponding content by driving the video processor162 and the audio processor 163.

The display 110 may display an image frame which is generated by thevideo processor 162.

The speaker 167 outputs audio data which is generated by the audioprocessor 163.

The button 164 may be a button which may be embodied as various types ofbuttons such as a mechanical button, a touch pad, a wheel, etc. formedon a certain area of the user terminal device 100, for example, a frontpart, a side part, or a rear part of the external portion of the mainbody of the user terminal device 100.

The microphone 165 is an element to receive a user voice or othersounds, and convert the same to audio data.

The controller 130 may use a user voice input through the microphone 165during a phone call, or convert the user voice to audio data and storethe same in the storage 140. The camera 166 is an element to photographa still image or a video according to a user control.

The camera 166 may be realized as a plurality of cameras such as a frontcamera and a rear camera. When the camera 166 and the microphone 165 areprovided, the controller 130 may perform a control operation accordingto user voice input through the microphone 165 or a user motionrecognized by the camera 166.

That is, the user terminal device 100 may operate in a motion controlmode or a voice control mode. When the user terminal device 100 operatesin a motion control mode, the controller 130 photographs a user byactivating the camera 166, traces a change in the user motion, andperform a corresponding control operation.

When the user terminal device 100 operates in a voice control mode, thecontroller 130 may operate in a voice recognition mode where a uservoice input through the microphone 165 is analyzed and a controloperation is performed according to the analyzed user voice. In the userterminal device 100 which supports a motion control mode or a voicecontrol mode, a voice recognition technology or a motion recognitiontechnology may be used in various exemplary embodiments. For example, ifa user makes a motion of drawing a circle which includes a plurality ofobjects, or if a user utters a plurality of objects, control operationscorresponding each of the plurality of objects may be performed.

The motion control technology or the voice control technology may alsobe applied in other exemplary embodiments. The movement detector 168 isan element to detect the motion of the main body of the user terminaldevice 100. That is, the user terminal device 100 may rotate or tilt invarious directions. The movement detector 168 may detect motionproperties such as rotation direction, angle, tilt, etc. by using atleast one of various sensors including a geomagnetic sensor, a gyrosensor, an acceleration sensor, etc.

In addition, albeit not illustrated in FIG. 31, the user terminal device100 may further include a USB port to be connected with a USB connector,various external input ports to be connected with various externalterminals such as a headset, a mouse, and a Digital MultimediaBroadcasting (DMB) chip to receive and process a DMB signal depending onexemplary embodiments.

The above various exemplary embodiments may be realized by a userterminal device 100 in various forms as described above. Meanwhile, theobjects of the above-described various exemplary embodiments may bevarious types of objects which are displayed on various types of screenssuch as a home screen, an icon screen, a list screen, an applicationexecution screen, a web browser screen, a content play screen, etc.

In particular, the home screen may be a basic screen in a fixed form ora user creative screen.

When the home screen is configured as a user creative screen, a user maydraw an object on the home screen arbitrarily and match at least onecontrol operation to the object.

Hereinafter, the method of configuring a user creative screen and itsform will be described in greater detail.

<Example of Configuring a User Creative Screen>

FIG. 32 is a view illustrating an example of a configuration of a usercreative screen which is generated directly by a user. According to FIG.32, various objects 2110˜2190 may be displayed on a home screen 2100 ofthe user terminal device 100. Each of the objects 2110˜2190 is generatedas the user draws it on the home screen 2100. Accordingly, the displayattributes each of the objects 2110˜2190 such as shape, location, color,etc. may be provided, not in a fixed form, but in various ways accordingto users. In this case, some of the objects 2120 may be drawn to includea plurality of objects 2121˜2124. In addition, a user may match anobject which he or she draws with a desired control operation.

When a user matches a control operation to each object, the controller130 stores the matching information in the storage 140 or in otherstorage means. Subsequently, when a displayed object is selected, thecontroller 130 may perform a control operation matched with the objectbased on the matching information.

As illustrated in FIG. 32, an object 2120 may be configured to include aplurality of objects 2121˜2124. In this case, even if a user selectsonly the corresponding object 2120, a plurality of control operationmatched with each of the objects 2121˜2124 may be performed. Inaddition, it is not necessary to match a control operation to all of theobjects, and some of the objects may be used only for visualsatisfaction.

FIG. 33 is a view illustrating another example of a user creativescreen. As illustrated in FIG. 33, the user terminal device 100 maydisplay the home screen 2210 when an unlock operation is performed whenthe user terminal device 100 is in a lock state. If the user terminaldevice 100 is an apparatus which can be controlled by a pen and anunlock operation is performed using a pen, a graphic effect which isspecific to a pen may be displayed. That is, as illustrated in FIG. 33,when a user takes a gesture of dividing a lock screen 220 with a pen,the controller 130 displays the home screen 2210 while providing agraphic effect of tearing the lock screen 2200 as if a paper is tornapart.

A plurality of objects 2211˜2219 which are produced previously by a usermay be disposed on the home screen 2210. A user may delete or changeexisting objects which are provided as a default by drawing them usinghis or her finger or a pen, or may add new objects.

In addition, a user may arbitrarily select a control operation which ismatched with each object so as to link an application, a specific image,a specific function, a bookmark, etc. with each object. Accordingly, theuser may realize a home screen which may represent the user's individualpersonality.

FIGS. 32 and 33 illustrate a case where a home screen is configured as auser creative screen, but a home screen is not limited thereto. Anobject which is drawn directly by a user on various UI screens such asan application icon screen, a lock screen, etc. may also be displayed.Meanwhile, a user may arbitrarily select properties given to the pen 200using an authoring tool.

FIG. 34 illustrates an example of a screen authoring tool. When anauthoring tool menu (not shown) is selected while the home screen 2210is displayed, the controller 130 may display an authoring tool screen2300 as illustrated in FIG. 34. According to FIG. 34, various tools toselect the color of line, the shape of object, the number of items to begenerated, a special symbol, etc. may be displayed on the authoring toolscreen 2300. When a user arbitrarily selects these tools, completessettings, and manipulates a screen using an input means, an object maybe drawn according to the set display properties. Accordingly, the usermay display more objects in more various ways.

Meanwhile, a home screen may be configured as a background screen and anobject. The background screen and the object may be disposed indifferent layers and displayed to be overlapped with each other. Herein,the object refers to various graphic images which are displayed on thehome screen, such as an icon, a widget, etc. A user may directly drawthe background screen itself or an object displayed on the backgroundscreen so as to generate his or her unique home screen.

FIGS. 35 to 37 are views provided to explain various exemplaryembodiments of displaying a home screen using a plurality of layers.According to FIG. 35, the display 110 displays a single home screen 3500composed the first to the third layers 3510, 3520, 3530. An image to bedisplayed on each layer 3510, 3520, 3530 is stored a graphic buffer (notshown) corresponding to each layer 3510, 3520, 3530 and provided to thedisplay 110.

In FIG. 35, a widget 3511 is displayed on the first layer 3510, andicons 3521, 3522 are displayed on the second layer 3520. The controller130 detects a boundary of each object displayed on the first layer 3510and the second layer 3520 and confirms the location of the object.Accordingly, as a touch coordinates value is detected from a touchsensor included in the detector 120, the touch coordinates value and thelocation of each object are compared, and it is determined whether anobject is selected or not. A background screen 3531 is displayed on thethird layer 3530.

A user may directly draw each object or a background screen displayed onthe first to the third layers 3510, 3520, 3530. The GPU 134 of thecontroller 130 applies the size, location, shape, thickness, etc. of thetrace which is drawn on the screen by a user by pixel unit, and renderseach object and the background screen on the corresponding layer.Accordingly, the object and the background screen may be used accordingto the user's intention. The display 110 displays each layer 3510, 3520,3530 to be overlapped with each other sequentially. Accordingly, theuser may recognize a single home screen consisting of a backgroundscreen 3530 and objects 3511, 3521, 3522.

In FIG. 35, the objects 3511, 3521, 3522 may be directly drawn on thefirst or second layer 3510, 3520 while the background screen 3530 isdisplayed. Otherwise, the objects 3511, 3521, 3522 may be drawn in aseparate program execution screen, and then, moved to the backgroundscreen 3530.

Further, only three layers are illustrated in FIG. 35, but the number oflayers is not limited thereto, and the disposition of the layers is alsonot limited to the order of FIG. 35. For example, the widget 3511 may bedisplayed on the second layer, and the icons 3521, 3522 may be displayedon the first layer. Also, the widget or the icons may be dispersed anddisplayed on a plurality of layers, respectively. Meanwhile, accordingto another exemplary embodiment, a user may draw a single backgroundscreen, divide the background screen into a plurality of areas, andactivate the areas just like objects.

FIG. 36 is a view provided to explain the method of configuring a homescreen according to another exemplary embodiment. According to FIG. 36,the display 110 displays a single home screen 3600 by overlapping thefirst to third layers 3610, 3620, 3630. The widget 3611 or other icons(not shown) from among each of the layers 3610, 3620, 3630 may bedisplayed, and a background screen 3631 may be displayed on the thirdlayer 3630. The second layer 3620 may be realized as a transparent layerfor user interaction. A user may draw a background screen 3631 on thethird layer 3630, divide some of the entire area of the backgroundscreen 3631 into a plurality of areas, and set the areas as independentchunk areas 3631-1˜3631-5. The chunk areas may be set manually by auser's selection, or may be set automatically by the user terminaldevice 100.

In case where the chunk areas are set manually, when a user draws onebackground screen 3631, the controller 130 stores the background screen3631 in the storage 140. Subsequently, when the user performs amanipulation of distinguishing each area by touching and dragging thearea using an input means such as a finger and a pen, the controller 130may display a graphic line according to the manipulation on the secondlayer 3620.

When such a manipulation is completed, the controller 130 recognizes aplurality of areas which are specified by a user trace drawn on thesecond layer 3620 as a plurality of transparent objects 3621-1˜3621-5,respectively. The controller 130 stores a pixel coordinates value ofeach of the plurality of transparent objects 3621-1˜3621-5. In addition,the controller 130 also matches a control operation with each of theplurality of transparent objects 3621-1˜3621-5 and store the matchinginformation. Subsequently, when various user manipulations such astouch, hovering, etc. are detected, the controller 130 confirms atransparent object having a pixel coordinates value corresponding to thepoint where the manipulations are detected, and performs a controloperation corresponding to the confirmed transparent object.

Meanwhile, in case where the chunks are set automatically, thecontroller 130 analyzes the characteristics of a user manipulation, anddivides the background screen 3631 into a plurality of areas.Specifically, the controller 130 may divide the screen according tovarious standards such as unit of closed curve, stroke, the number ofstrokes, the order of drawing, etc.

The unit of closed curve refers to recognizing a portion constituting aclosed curve in one object as an independent chunk, and the unit ofstroke means recognizing a portion which is drawn from when the pen 200starts touching a home screen until when the pen 200 finishes touchingthe home screen as one chunk. The unit of the number of strokes refersto recognizing a portion which is drawn whenever there is a stroke untilthe number of stores reaches a predetermined number of times as anindependent chunk, and the unit of the order of drawing refers todistinguishing a portion which is drawn first with reference to apredetermined time unit and a portion which is drawn later andrecognizing each portion as an independent chunk. In addition, thecontroller 130 may divide objects according to various standards.

When distinction between chunks is made, the controller 130 assumes thatthere are virtual transparent objects 3621-1˜3621-5 corresponding toeach chunk, and stores a pixel coordinates value of the objects3621-1˜3621-5. In addition, the controller 130 stores informationregarding a control operation matched with each of the objects3621-1˜3621-5. Accordingly, when a user selects one of the plurality ofareas 3631-1˜3631-5 displayed on the background screen 3611 on the homescreen 3600, the controller 130 performs a control operation matchedwith the selected area.

According to an exemplary embodiment of FIG. 36, a user may draw a homescreen in a desired size and shape without any limit in the size of anobject, and may match a control operation with a desired area within thehome screen.

For example, a user may match a desired control operation even with abasis area where there is no sign or object within the background screen3631. That is, a user may set even a basis area which cannot berecognized by a third party as a transparent object and use the area.

In the exemplary embodiment of FIG. 36, the number of layers and theorder of disposition may also vary as described above.

Meanwhile, FIGS. 35 and 36 describes an exemplary embodiment where auser draws the background screen of the home screen directly, butaccording to another exemplary embodiment, a user may overlap a widgetscreen which the user draws directly with the front of the existing homescreen as if the user changes the home screen itself.

FIG. 37 is a view provided to explain a method for configuring a homescreen according to the above exemplary embodiment. According to FIG.37, a widget screen 3711 is displayed on the uppermost layer 3710. Auser may draw the widget screen 3711 directly as he or she wants. Thewidget screen 3711 is displayed on another lower layer 3720 in anoverlapping manner. Accordingly, the user may use the widget screen 3711as if it is the home screen 3700.

As described in FIG. 36, the widget screen 3711 of FIG. 37 is alsodivided into a plurality of chunk areas 3711-1˜3711-5, and each chunkarea 3711-1˜3711-5 may be used as if it is an object. In addition,another transparent layer may further added on top of the first layer3710 where the widget screen 3711 is displayed. The method ofdistinguishing chunks has already described above with reference to FIG.36 and thus, further description will not be provided.

The controller 130 may display the existing home screen on one 3720 ofsub layers of the first layer 3710. The controller 130 may change thedisplay order of the layer 3720 where the existing home screen isdisplayed and the layer 3710 where the widget screen is displayedaccording to a user manipulation, and directly provide the existing homescreen to a user. That is, according to an exemplary embodiment, theuser terminal device 100 may provide a user creative screen using awidget program without changing the existing home screen. Meanwhile, asdescribed above, at least one control operation may be matched with eachof the various objects (including transparent objects) included in thehome screen. A user may draw the home screen directly or an object onthe home screen, and select a control operation to be matched with.

FIGS. 38 to 42 describe various methods of generating an object to bematched with a control operation in greater detail.

FIG. 38 is a view illustrating an example of a home screen generatingmethod. According to FIG. 38, a user may draw various objects 2410, 2420using the pen 200 on the screen 2400 as if the home screen 2400 is acanvas.

After drawing the objects 2410, 2420, the user may select a controloperation to be matched with the objects. When a user command to match acontrol operation is input, the controller 130 displays a list ofapplications installed in the user terminal device 100 for each of theobjects 2410, 2420. In FIG. 38, the user terminal device 100 includes ahome button 101 and a hidden button 102 at a bezel portion, and when thehidden button 102 is selected, each application list 2411, 2421 isillustrated, but a user command may be input in various ways.

The user may select an application to be matched with each object on theapplication lists 2411, 2421. When the selection is completed, thecontroller 130 may match information regarding the corresponding objectwith information regarding the selected application and store thematching information in the storage 140 or other storage means.Accordingly, a user creative screen may be generated.

FIG. 39 is a view provided to explain another example of a user creativescreen generating method. According to FIG. 39, when a user selects thehidden key 102 while a home screen 2500 is displayed, the controller 130displays various menus 2510 on one side of the home screen. The menusmay include a menu 2511 to generate an object directly.

When the user selects the menu 2511, the controller 130 may displayinformation regarding various applications installed in the userterminal device 100 in the form of list 2520. When the user selects oneapplication in this state, the home screen 2500 where the user can drawan object is displayed. In this case, a guide message 2530 which leadsthe user to draw an object may be displayed.

The user may draw an object 2540 on the home screen 2500 with referenceto the guide message 2530. When the object is drawn, the controller 130matches the object 2540 with an application which is previously selectedby the user, and displays the object 2540 on the final home screen 2500.The controller 130 may determine whether object-drawing is completed invarious ways. For example, if a predetermined time elapses after theuser inputs the last stroke to draw the object, the controller 130 maydetermine that object-drawing is completed. Alternatively, if a specificconfirmation button is pressed after an object is drawn, the controller130 may determine that object-drawing is completed. Afterwards, unlessdeleted or edited, the object 2540 is displayed on the home screen 2500in a fixed manner.

Meanwhile, if an application is selected first, a user may generate anobject by using a reference object which is matched with theapplication.

FIG. 40 is a view provided to explain a method of generating an objectby editing a reference object. As illustrated in FIG. 40, when a userselects an application first, a reference object 2610 corresponding tothe selected application is displayed first (a) of FIG. 40.

The user may edit the reference object 2610 using the pen 200 ((b) and(c) of FIG. 40). That is, as illustrated in FIG. 40, the user maygenerate an object 2610 in the shape of star by touching each edge ofthe reference object 2610 and stretches it in various directions usingthe pen 200 ((d) of FIG. 40).

FIG. 40 illustrates only the case of changing the shape, but the usermay also generate a whole new object by changing the size, color,attached text, etc. of the reference object 2610. In this case, anauthoring tool may be used as illustrated in FIG. 34.

FIG. 41 illustrates an exemplary embodiment where a guide image isdisplayed so that a user may draw an object more easily.

According to FIG. 41, when a user selects an object generating menu, thecontroller 130 displays an application list 2710 on the home screen2700. The user may select an application which he or she wants to matchwith an object to be drawn first on the application list 2710. In thiscase, the application list 2710 disappears, and a reference object 2720corresponding to the selected application is displayed on the homescreen in the form of watermark. The reference object 2720 may bedisplayed in a dotted form so that the user may draw it easily followingthe dots. The user may draw the object 2730 following the line of theobject 2720. In this case, it is not necessary that the user draws theobject exactly the same as the reference object 2720. Finally, when theobject 2730 is completed, the controller 130 matches an applicationwhich he or she has selected with the completed object 2730.

Meanwhile, even though the user does not select a control operation byhimself, the user terminal device 100 may recognize an object drawn bythe user and recommend an object that is suitable for the object.

FIG. 42 is a view provided to explain the method of recommending anapplication corresponding to an object. According to FIG. 42, when auser draws an object 2810 on a home screen 2800, the controller 130recognizes the characteristics of the object 2810. Specifically, thecontroller 130 may recognize the shape of the drawn object 2810 usingthe above-described handwriting recognizing module 143-9.

The controller 130 may compare the icons or application titles of eachapplication which is pre-installed with the recognized shape, and searchan application which has a similarity that is higher than apredetermined value. For example, if a user draws an object in the shapeof bird, the controller 130 may search an application which has a birdshape as a basic icon.

In addition, the controller 130 may search an application which providesa service related to a recognized shape. For example, if a draws anobject in the shape of sun as illustrated in FIG. 42, the controller 130may search an application related to weather.

The controller 130 may recommend applications to a user. That is, thecontroller 130 may display at least one list 2820 of applications on oneside of the object 2810. If a user selects one application from the list2820, the selected application may be matched with an object.

As described above, a user may configure a user creative screen invarious ways. Since the user draws an object and configures a usercreative screen in person, the user may easily remember which object islocated in which area of the user creative screen. That is, even thoughmany small objects are displayed on a single page, the user mayintuitively know the location of each object. Accordingly, the userterminal device 100 may display many small objects on a single pagewithout configuring the page of a home screen as a plurality of pages.In this case, the user may magnify the small objects on the single pagethrough a zoom-in or zoom-out manipulation.

FIG. 43 is a view provided to explain the method of performing a zoom-inor zoom-out manipulation with respect to a user creative screen.According to FIG. 43, various objects are displayed on the user creativescreen 2900. Some of the objects 2910˜2960 may be displayed in such asmall size that a user may not discern their shape easily. In thisstate, the user may perform a zoom-in operation by tapping the partwhere the reduced objects 2910˜2960 are displayed or touching the screenwith a plurality of fingers and widening the touched screen. In thiscase, the whole home screen 2900 is magnified, and the reduced objects2910˜2960 are enlarged and displayed. The user may select one of theenlarged objects and executed a matched control operation.

Meanwhile, if the user taps a black area where no objects are displayed,or performs the manipulation of touching the screen with a plurality offingers and narrowing the touched screen, the controller 130 performs azoom-out operation. Accordingly, the size of the screen goes back to itsoriginal state, and each of the objects is displayed in the reducedsize. Thus, a large number of objects may be displayed on a singlescreen without adding pages.

In addition, the user terminal device 100 may realize a user creativescreen in the form of roll screen. That is, the user terminal device 100may touch and drag the screen in one direction while the user creativescreen is displayed so that the screen may be scrolled gradually alongthe dragging direction.

FIG. 44 is a view provided to explain an exemplary embodiment where ahome screen is changed using a scroll method. According to FIG. 44, thecontroller 130 displays a home screen 3000 including various objects. Inthis state, when a user touches the screen using a finger or a pen andscrolls the screen in an arrow direction, the screen moves and an emptyspace 3010 newly appears. The user may utilize the empty space 3010 bydrawing a new object in the empty space 3010 and matching a controloperation. FIG. 44 illustrates a case where the home screen is scrolledcontinuously in an up-down direction, but the direction of scrolling isnot limited thereto and may be realized in various directions.

When there is scrolling, the controller 130 may provide a graphic effectof waving the home screen by the scrolling, or may also provide otheradditional feedbacks such as sound, vibration, light, etc.

As described above, the user terminal device 100 may provide a homescreen including objects which are generated directly by a user, thatis, a user creative screen. The shape, size, color, etc. of the objectson the user creative screen may be realized in various ways. Inparticular, a user may easily set the location of the objects.

Accordingly, a user may conveniently dispose frequently-used objects,select a plurality of objects simultaneously when necessary, and executea control operation corresponding to each object simultaneously. Themethod of selecting objects simultaneously may be at least one of themanipulation of drawing a closed curve including a plurality of objects,the manipulation of drawing a connection line between a plurality ofobjects, the multi-touch manipulation of touching each of a plurality ofobjects, and the manipulation of covering an area where a plurality ofobjects are displayed as described above.

In addition, when a user generates an object, the user may select aplurality of control operations which are to be matched with the object,or may set different control operations according to the user'sselection pattern regarding the object such as a selected direction or aselected point. Thus, a control method according to the above-describedvarious exemplary embodiments may be realized on the user creativescreen.

As described above, according to various exemplary embodiments, a usermay easily select one or a plurality of control operationssimultaneously by selecting objects.

The method of controlling a user terminal device or the method ofgenerating a user creative screen according to the above-describedvarious exemplary embodiments may be coded as software and stored in anon-transitory readable medium. Such a non-transitory readable mediummay be mounted and used in various apparatuses.

Specifically, a program code to perform a controlling method includingthe steps of displaying a home screen, and when a user manipulation ofassociating a plurality of objects included in the home screen isdetected, performing a control operation corresponding to each of theplurality of objects may be stored in a non-transitory readable mediumand provided therein.

The non-transitory recordable medium refers to a medium which may storedata semi-permanently and which may be readable by an apparatus.Specifically, the above-mentioned various applications or programs maybe stored in a non-transitory recordable medium such as a CD, a DVD, ahard disk, a Blu-ray disk, a USB, a memory card, and a ROM and providedtherein.

The foregoing exemplary embodiments and advantages are merely exemplaryand are not to be construed as limiting. The present teaching can bereadily applied to other types of apparatuses. Also, the description ofthe exemplary embodiments is intended to be illustrative, and not tolimit the scope of the inventive concept, as defined by the appendedclaims, and many alternatives, modifications, and variations will beapparent to those skilled in the art.

What is claimed is:
 1. A user terminal device, comprising: a displayconfigured to display a screen including an object; a detectorconfigured to detect a user drawing input which is input on the objector in an area surrounding the object; a controller configured to, inresponse to the user drawing input being detected, execute a programcorresponding to the object and display, on the display, a screen wherethe user drawing input is processed as an input of the program.
 2. Theuser terminal device as claimed in claim 1, wherein the controller isfurther configured to, in response to the user drawing input comprisingtext, display, on the display, a screen where the text is processed asan input of the program.
 3. The user terminal device as claimed in claim2, wherein the controller is further configured to, in response to theobject comprising a search widget, perform a search regarding the textand display, on the display, a result of the search.
 4. The userterminal device as claimed in claim 1, wherein the controller is furtherconfigured to, in response to the user drawing input comprising asymbol, display, on the display, a screen where the symbol is processedas an input of the program.
 5. The user terminal device as claimed inclaim 2, wherein the controller is further configured to, in response tothe object being an icon matched with a mail program, display, on thedisplay, a mail writing screen to transmit mail to a mail addresscorresponding to the text.
 6. The user terminal device as claimed inclaim 2, wherein the controller is further configured to, in response tothe object being an icon matched with a messenger program, display, onthe display, messenger contents which are exchanged with another usercorresponding to the text.
 7. The user terminal device as claimed inclaim 1, wherein the user drawing input is input through atouch-and-drag manipulation using a user's body or a pen.
 8. A userterminal device, comprising: a display configured to display a screenincluding a plurality of objects; a detector configured to detect a usermanipulation on the screen; and a controller configured to, in responseto a connection line between a first object and a second object fromamong the plurality of objects being drawn according to the usermanipulation, perform a first control operation corresponding to thefirst object and a second control operation corresponding to the secondobject in association with each other.
 9. The user terminal device asclaimed in claim 8, wherein the controller is further configured todisplay a first screen by executing a first program corresponding to thefirst object, and in response to a user manipulation being performed onthe first screen, execute a second program corresponding to the secondobject and display a second screen which reflects a result according tothe user manipulation.
 10. The user terminal device as claimed in claim9, wherein the controller is further configured to determine a contentwhich is displayed on the first screen according to a type of the secondprogram.
 11. A controlling method of a user terminal device, comprising:displaying a first screen including an object; detecting a user drawinginput which is input on the object or in an area surrounding the object;and executing, in response to the user drawing input being detected, aprogram corresponding to the object and displaying a second screen wherethe user drawing input is processed as an input of the program.
 12. Thecontrolling method as claimed in claim 11, wherein the displaying thesecond screen comprises, in response to the user drawing inputcomprising text, displaying a screen where the text is processed as aninput of the program.
 13. The controlling method as claimed in claim 12,wherein the displaying the second screen comprises, in response to theobject being a search widget, performing a search regarding the text anddisplaying a result of the search.
 14. The controlling method as claimedin claim 11, wherein the displaying the second screen comprises, inresponse to the user drawing input comprising a symbol, displaying ascreen where the symbol is processed as an input of the program.
 15. Thecontrolling method as claimed in claim 12, wherein the displaying thesecond screen comprises, in response to the object being an icon matchedwith a mail program, displaying a mail writing screen to transmit mailto a mail address corresponding to the text.
 16. The controlling methodas claimed in claim 12, wherein the displaying the second screencomprises, in response to the object being an icon matched with amessenger program, displaying messenger contents which are exchangedwith another user corresponding to the text.
 17. The controlling methodas claimed in claim 12, wherein the displaying the second screen furthercomprises: in response to the user drawing input being detected while apredetermined condition is satisfied, displaying the second screenaccording to the user drawing input; and in response to the user drawinginput being detected while the predetermined condition is not satisfied,changing the first screen by page unit according to a direction of theuser drawing input.
 18. A controlling method of a user terminal device,comprising: displaying a screen including a plurality of objects;detecting a user manipulation on the screen; and in response to aconnection line between a first object and a second object from amongthe plurality of objects being drawn according to the user manipulation,performing a first control operation corresponding to the first objectand a second control operation corresponding to the second object inassociation with each other.
 19. The controlling method as claimed inclaim 18, wherein the performing comprises: displaying a first screen byexecuting a first program corresponding to the first object; andexecuting, in response to a separate user manipulation being performedon the first screen, a second program corresponding to the second objectand displaying a second screen which reflects a result according to theseparate user manipulation of the first screen.
 20. The controllingmethod as claimed in claim 19, wherein the performing further comprises:determining a content which is displayed on the first screen accordingto a type of the second program.
 21. A user terminal device, comprising:a display configured to display a screen including an object; a detectorconfigured to detect a user manipulation on the screen; and a controllerconfigured to, in response to a first area of the object being selectedaccording to the user manipulation, perform a first control operationrelated to the object, and in response to a second area of the objectbeing selected, perform a second control operation related to theobject.